Types of Neural Network

Computer Vision

• Image Classification

Computer Vision

• Object Detection

Computer Vision

• Neural Style Transfer

Is it math?

Image Data

Image Data

Convolutions

Edge Detection

• Vertical Edge Detection

Parameters

Padding

• Pad so that output size is the same as the input size.

Strided convolutions

• \(n\times n\) (\(n=7\))
• \(f \times f\) (\(f=3\))
• no padding
• stride \(s=2\)

Summary of Convolutions

• \(n \times n\) image; \(f \times f\) filter
• padding \(p\); stride \(s\)
• Output: \(\left[ \frac{n+2p-f}{s}+1 \right] \times \left[ \frac{n+2p-f}{s} +1 \right]\)

Convolutions Over Volume

• https://www.youtube.com/watch?v=HzuFgLnhgqw

Your Turn: Number of Parameters in One Layer

Question: If you have 10 filters that are \(3 \times 3 \times 3\) in one layer of a neural network, how many parameters does that layer have?

Summary of Notation

If layer \(l\) is a convolution layer:

• \(f^{[l]}\) = filter size
• \(p^{[l]}\) = padding
• \(s^{[l]}\) = stride
• \(n^{[l]}_C\) = number of filters
• Each filter: \(f^{[l]} \times f^{[l]} \times n^{[l-1]}_C\)
• Activations: \(a^{[l]} \rightarrow n_H^{[l]} \times n_W^{[l]} \times n_C^{[l]}\)
• Weights: \(f^{[l]} \times f^{[l]} \times n^{[l-1]}_C \times n^{[l]}_C\)
• bias: \(b^{[l]}_C\)
• Input: \(n^{[l-1]}_H \times n^{[l-1]}_W \times n^{[l-1]}_C\)
• Output: \(n^{[l]}_H \times n^{[l]}_W \times n^{[l]}_C\)
• \(n^{[l]}_H = \frac{n^{[l-1]}_W + 2p^{[l]}-f^{[l]}}{s^{[l]}}+1\) (similar for \(W\))

Pooling Layers

Pooling Layers

Examples: LeNet - 5

LeCun et al., 1998. Gradient-based learning applied to document recognition

Examples: LeNet - 5

Types of Layer in A Convolutional Network

• Convolution

• Pooling

• Fully Connected

Using Keras To Build CNN

Typical keras workflow:

1. Define your training data: input tensors and target tensors
2. Define a network of layers (or models) that maps your inputs to your targets
3. Configure the learning process by choosing a loss function, an optimizer, and some metrics to monitor
4. Iterate on your training data by calling the fit() method of your model

Using Keras To Build CNN

# Define model structure
cnn_model <- keras_model_sequential() %>%
  layer_conv_2d(filters = 32, kernel_size = c(3, 3), 
  activation = "relu", input_shape = input_shape) %>%
  layer_max_pooling_2d(pool_size = c(2, 2)) %>%
  layer_conv_2d(filters = 64, kernel_size = c(3, 3), activation = "relu") %>%
  layer_dropout(rate = 0.25) %>%
  layer_flatten() %>%
  layer_dense(units = 128, activation = "relu") %>%
  layer_dropout(rate = 0.5) %>%
  layer_dense(units = num_classes, activation = "softmax")

Using Keras To Build CNN

• Compile and define loss function, optimizer and metrics to monitor during the training

# Compile model
cnn_model %>% compile(
  loss = loss_categorical_crossentropy,
  optimizer = optimizer_adadelta(),
  metrics = c('accuracy')
)

Using Keras To Build CNN

• Fit the model using training dataset and define epochs, batch size and validation data

# Train model
cnn_history <- cnn_model %>%
  fit(
    x_train, y_train,
    batch_size = batch_size,
    epochs = epochs,
    validation_split = 0.2
  )

• Predict new outcomes using the trained model

# Model prediction
cnn_pred <- cnn_model %>%
  predict_classes(x_test)

Size of the Model

Effective CNNs

• LeNet -5: LeCun et al., 1998. Gradient-based learning applied to document recognition

• AlexNet: Krizhevsky et al., 2012. ImageNet Classification with Deep Convolutional Neural Networks

• VGG-16: Simonyan & Zisserman 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition

• ResNets: He et al., 2015. Deep Residual Learning for Image Recognition

Different Architecture Search Algorithms:

• NASnet: 1800 GPU days (5 yrs on 1 GPU)

• AmoebaNet: 3150 GPU days

• DARTS: 4 GPU days

• ENAS: 1000 x cheaper than standard NAS

Understanding Neural Networks

• Understanding Neural Networks Through Deep Visualization, Video

• Deep Neural Networks are Easily Fooled, Video